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ABSTRACT:  The  United  States  Air  Eorce  is  positioned  to  take  full  advantage  of  Internet2 
technologies  and  apply  them  to  Command  and  Control  (C2)  research.  This  paper  summarizes 
and  updates  progress  on  the  Air  Eorce’ s  Distributed  Mission  Training  Research  Network  (DMT- 
Rnet),  an  Intemet2  based  network  for  collaborative  research  and  training  via  distributed  PC- 
based  systems.  This  network  hosts  complex  environments  for  multi-operator  simulation-based 
training  and  performance  research. 

INTRODUCTION 

In  the  Air  Eorce  Research  Eaboratory,  a  major  research  focus  is  the  investigation  and 
enhancement  of  operational  expert  training  through  a  classified  SIMNET-based  infrastructure  to 
enhance  Distributed  Mission  Training  (DMT).  The  USAE  DMT  program  is  national  in  scope. 
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with  the  goal  of  enhancing  operational  training  through  high-fidelity  military  simulation  systems 
that  are  networked  using  secure,  classified  systems. 


DMT-RNet 

In  contrast,  the  DMT-Research  Net  (DMT-RNet)  project  is  a  local  project  that  will 
support  highly  controlled  and  experimental  basic  research  to  support  DMT  through  PC-based 
systems  networked  through  the  Internet.  The  project  will  establish  an  1-2  based  infrastructure  for 
collaborative  research  and  training  and  identify  specific  research  issues  related  to  enhanced  skill 
acquisition  and  operational  performance.  This  research  will  guide  improvements  in  the 
operational  USAF  DMT  training  environment  (Barnes,  Elliott  &  Entin,  2001). 

The  USAE  DMT  project  relies  on  a  network  of  highly  realistic  battle  simulators  that 
allow  expert  operators  to  train  in  a  virtual  battlespace  across  a  highly  secure  and  classified 
communication  network.  In  contrast,  DMT-Rnet  systems  enable  systematic  investigations  in 
unclassified  mode  and  establish  the  infrastructure  to  conduct  multi-level  investigations  of 
operational  performance  using  less  costly  PC-based  systems.  The  DMT-Rnet  collaborative 
research  program  will  leverage  emerging  1-2  capabilities  to  connect  distributed  PC -based 
simulation  systems  and  create  complex  environments  for  multi-operator  training  and 
performance  research.  These  systems  can  be  readily  deployed  to  operational  field  settings  and 
enable  cost-effective  distributed  training  wherever  Internet  access  is  available. 

DMT-Rnet  systems  will  not  be  total  replications  of  operational  systems.  Instead,  these 
synthetic  team  task  sytems  will  capture  the  cognitive  and  task  demands  of  most  interest  to 
trainers  and  researchers  (Elliott,  Dalrymple,  Regian.,  &  Schiflett,  2001).  Convincing  arguments 
have  been  stated  for  the  relevance  of  systems  which  are  based  on  psychological  fidelity,  and  the 
absolute  need  for  internal  validity  for  the  advancement  of  scientific  knowledge  (Berkowitz  & 
Donnerstein,  1982;  Cook  &  Campbell,  1979;  Mook,  1983).  Eor  example,  a  PC-based  system 
may  simplify  the  “button  pressing”  procedures  required  in  an  actual  operational  system  and 
instead  focus  on  display  characteristics,  decision  making  processes,  tactics,  strategies,  and/or 
teamwork  functionsinitial  efforts  developed  PC-based  platforms  to  represent  the  underlying 
cognitive  and  decision  making  task  demands  of  Airborne  Warning  and  Control  System 
(AW ACS)  Weapons  Director  Teams,  based  on  multiple  investigations  of  cognitive  and 
functional  aspects  of  this  performance  domain  (Coovert,  et  ah,  1999). 

The  DMT-Rnet  also  has  the  capabilities  for  integrating  other  PC -based  simulators  into  its 
network,  making  possible  a  simulator  interoperability  that  previously  did  not  exist.  This  allows 
for  flexible  options  of  utilization  of  the  network,  with  different  levels  of  fidelity  and 
classifications  involved. 

As  a  pioneering  technology,  DMT-Rnet  enables  distributed  and  multidiscipline 
collaborations  toward  complementary  research  goals.  Universities,  agencies,  and  companies  will 
be  able  to  utilize  a  common  platform,  collected  data  in  distributed  settings,  and  pool  their  talents 
and  resources  to  produce  high  level  research.  The  applications  of  these  concepts  and 
technologies  to  other  realms  are  nearly  limitless,  given  imagination  and  initiative. 

The  DMT-Rnet  is  an  ideal  technology  jump  for  C2  research.  It  facilitates  the  gathering 
of  information  on  resource  allocation  in  the  various  C2  roles  as  well  as  how  data  is  shared 
throughout  the  C2  levels.  Barnes,  Elliott,  and  Entin  (2001)  give  an  example  where  someone  is 
playing  the  role  of  an  Intelligence,  Surveillance  and  Reconnaissance  (ISR)  operator.  This  person 
has  control  of  various  information-gathering  tools,  such  as  Predator  Uninhabited  Aerial  Vehicles 


(UAVs).  The  operator  may  send  out  these  Predators  to  find  specific  targets  or  locations.  Once 
found,  the  ISR  operator  sends  this  information  to  the  person  needing  that  information.  It  can  be 
sent  to  everyone  in  his  or  her  particular  site,  or  to  other  individuals  requiring  that  specific  piece 
of  information.  As  an  example,  someone  playing  a  role  in  the  Naval  C2  team  at  the  University 
of  South  Florida  node  can  send  a  message  to  the  person  playing  the  role  of  the  Joint  Forces  Air 
Component  Commander  at  Brooks  Air  Force  Base  to  request  control  of  a  KC-135  (refueling 
tanker).  The  JFACC  can  then  switch  control  of  the  KC-135  to  his  naval  counterpart  working 
from  the  University  of  South  Florida.  Communication  and  resource  allocation  within  and 
between  the  different  sites  can  be  closely  monitored  and  documented.  Analyses  of 
communication  and  coordination  data  will  reap  enormous  amounts  of  useful  information  on 
command,  control,  and  communications  process  measures. 

Demonstration  of  Distribution:  Single  Platform  Example 

The  initial  phase  of  the  DMT-Rnet  project  utilized  the  dynamic  distributed  decision 
making  (DDD)  team-in-the-loop  simulation  environment  (Hess,  MacMillan,  Elliott,  &  Schiflett, 
1999;  Kleinman  and  Serfaty  1989).  We  developed  an  internet-based  version  of  the  DDD,  the 
DDD  Network  (DDDnet),  which  allows  players  in  distributed  locations  to  connect  and  perform  a 
distributed  mission  in  real  time  (Barnes,  Elliott  &  Entin,  2001).  The  DDDnet  is  an  internet-ready 
version  of  a  Einux-based  collaborative  gaming  space  that  connects  players  to  each  other  and  to 
others,  such  as  observers,  confederates,  trainers,  or  researchers.  In  the  DDDnet  observers  at  any 
location  in  the  network  are  able  to  observe  the  scenario  play  in  real  time.  They  can  view  the 
screen  display  and  electronic  communications  of  any  player,  and  communicate  to  one  another  via 
email  or  voice.  In  addition,  the  DDDnet  can  connect  players  to  one  another  for  interactive 
mission  planning,  debriefings  and  after-action  reviews.  Other  internet-based  systems  have  also 
been  developed  for  specific  training  and  research  functions,  within  the  scope  of  the  DDD-Rnet 
project. 

DDD  simulations  in  general  are  based  on  broad  command  and  control  (C2)  functions  and 
have  been  demonstrated  to  elicit  important  team-oriented  cognitive  processes  such  as 
communication  and  coordination,  resource  allocation  and  sharing,  and  decision  making.  ,  Eor 
this  initial  effort,  the  DDD  software  and  scenarios  were  developed  as  analogues  to  USAE 
operational  performance  domains.  Specifically,  this  version  of  the  DDDnet  was  developed  to 
represent  the  underlying  cognitive  and  decision  making  task  demands  of  Airborne  Warning  and 
Control  System  (AW ACS)  Weapons  Director  Teams,  based  on  multiple  investigations  of 
cognitive  and  functional  aspects  of  this  performance  domain  (Coo vert,  et  ah,  2001).  Eurther 
development  resulted  in  a  scenario  that  emulates  three  military  C2  teams:  the  USAE  AW ACS 
team,  another  USAE  ground-based  C2  team,  and  a  third  Navy  airborne  C2  team. 

The  AW  ACS  DDD-Net  was  implemented  and  demonstrated,  allowing  distributed 
simulations  over  the  Internet.  Aptima,  a  small  research  corporation,  worked  with  faculty  and 
staff  at  the  University  of  Central  Elorida  (and  associated  Institute  for  Simulation  Technologies 
[UCE/IST]),  the  University  of  South  Elorida,  and  researchers  located  at  Brooks  Air  Eorce  Base  to 
test  the  feasibility  of  the  project  (Entin,  Serfaty,  Elliott,  &  Schiflett,  2001).  It  linked  the  different 
locations,  allowed  multi-role  missions,  data  collection,  and  feedback.  Different  parts  of  the 
network  included  T2  connections  for  improved  speed  and  performance.  The  DDDnet  achieved 
and  maintained  a  synchronized  connection  for  an  AW ACS  simulation  involving  16  participants. 


Simultaneously,  observers  at  each  location  rated  performance  using  web-based  tools  that  allowed 
immediate  data  pooling,  analysis,  and  feedback,  within  10  minutes  after  data  input  was  complete. 

Interoperability:  Connecting  Different  Platforms 

The  next  effort  of  this  project  extended  the  interoperability  capabilities,  as  well  as 
developing  an  additional  platform.  While  the  previous  effort  connected  several  nodes  located  at 
various  sites,  the  nodes  were  extensions  of  the  same  (DDD)  platform.  This  next  phase  will 
connect  several  different  platforms.  The  AW ACS  AEDGE™  (Agent-Enabled  Decision  Group 
Environment)  is  a  simulation  of  the  weapons  director  roles  within  the  Airborne  Warning  and 
Control  System.  The  C3STARS  (Command,  Control  and  Communications  Simulation  Training 
and  Research  System)  facility  is  a  high-fidelity  representation  of  C3  functions.  A  UAV 
platform  was  also  be  connected,  one  developed  at  New  Mexico  State  University,  and  a  cadre  of 
fighter-pilot  simulations,  using  the  4-ship  E-PASS  (Elight  Performance  Assessment  Simulation 
System).  The  following  figure  represents  the  participating  agencies,  locations,  and  platforms 
involved  in  this  effort. 
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AEDGE.  The  AEDGE  platform  (Agent-Enabled  Decision  Group  Environment)  is  a 
highly  configurable  C3  platform  that  uses  intelligent  agent  technology  to  enhance  simulation 
realism,  decision  support,  and  experimental  manipulations.  A  unique  aspect  to  the  agent-based 
task  is  the  capability  of  agents  to  “play”  any  role  in  a  given  scenario,  thus  allowing  the  study  of 
individual  performance  in  a  complex  but  controlled  team  setting.  Another  aspect  is  voice 
recognition  and  response,  where  human  voice  commands  direct  simulation  tasks,  and  agent- 
based  communications  can  be  heard. 


The  AEDGE  was  conceived  through  cognitive  and  functional  analysis  of  team  member 
roles,  responsibilities,  and  decisionmaking  (Chaiken  et  al,  2001),  to  optimize  generalizability  of 
results  to  operational  settings.  Systematic  descriptions  of  AW ACS  roles,  responsibilities, 
requirements,  interdependencies,  tactics,  strategies,  and  task  demands  were  collected  from 
subject  matter  experts,  cognitive  task  analyses  (Eahey  et  ah,  1998;  MacMillan  et  ah,  1998)  and 
focal-group  interviews  (Elliott  et  ah,  2001).  These  data  were  examined  to  identify  decision 
events,  which  were  generic  to  performance,  regardless  of  mission  scenario,  and  likely  to 
bottleneck  under  high  tempo  situations. 

The  software,  built  using  2E‘  Century  Systems  Inc.’s  AEDGE  infrastructure,  is  a 
distributed,  real-time  team  decision  support  environment  comprised  of  simulators,  entity 
framework,  intelligent  agents  and  user  interfaces.  The  environment  supports  a  wide  variety  of 
air,  sea  (surface  and  sub-surface),  and  ground  assets  in  a  combat  environment  (Chiara  &  Stoyen, 
1997),  primarily  based  on  the  roles  and  responsibilities  of  AW ACS  WD  team  members.  The 
environment  has  been  tested  with  an  excess  of  two  hundred  physical  entities  (planes,  ships,  SAM 
sites,  etc.)  operating  with  realistic  yet  non-classified  performance  characteristics  in  an  interactive 
environment  in  which  real-time  decision  support  is  available  to  each  WD. 

The  behavior  and  decisionmaking  of  all  hostile  and  friendly  entities  not  controlled  by 
humans  is  directed  by  agent-based  technology.  If  a  human  decides  to  “log  in”  as  a  particular 
entity,  he/she  may  choose  to  view  recommendations  generated  by  the  agent  for  that  entity.  Even 
if  the  human  operator  chooses  not  to  view  recommendations,  the  agent  recommendations  are  still 
logged  by  the  computer.  This  enables  direct  comparison  of  human  to  agent  decisionmaking. 
We  expect  these  capabilities  will  facilitate  skill  acquisition,  decision  making,  skills  assessment, 
and  human/team  performance  modeling. 

AEDGE  agent  capabilities  enable  more  detailed  and  innovative  approaches  to 
measurement  and  modeling  of  individual  and  team  workload,  communication  and 
decisionmaking.  Tracking  the  number  and  type  of  recommendations  generated  by  the  agent  at 
any  given  time  contributes  toward  new  ways  of  conceptualizing  and  representing  cognitive 
workload  of  individuals  and  teams.  Agent-based  recommendations  may  also  serve  as  a 
standardized  benchmark  by  which  human  tactics  and  decisions  can  be  compared.  In  addition,  the 
AEDGE  platform  can  operate  through  speech  -  operators  can  speak  to  the  system  using 
predefined  jargon,  request  tasks  be  performed  or  information  provided/transferred,  and  the 
agents  will  respond  verbally  to  the  speech-driven  requests,  using  voice  generation  technology. 
All  agent  communications  to  each  other,  as  well  as  to  humans,  are  transcribed,  logged  to  data 
output  files,  and  are  available  online. 


UAV  A  project  at  New  Mexico  State  University  (Cooke  &  Shope,  2002)  deals  with  such 
technology  in  the  realm  of  Uninhabited  Aerial  Vehicles  (UAVs).  The  UAV  Synthetic  Task 
Environment  (STE)  was  the  first  synthetic  task  designed  for  the  Cognitive  Engineering  Research 
on  Team  Tasks  (CERTT)  Eaboratory.  The  CERTT  Tab’s  hardware  design,  development,  and 
construction  phases  were  being  done  in  parallel  with  the  UAV  STE  development.  This  permitted 
close  developmental  cooperation  between  this  particular  STE  and  the  overall  CERTT  Tab 
design. 

Cooke  and  Shope  (In  Review)  briefly  summarize  the  UAV  is  controlled  by  operators  in  a 
GCS  (ground  control  station)  who  communicate  with  other  groups  concerning  issues  of  data 
interpretation  and  airspace  deconfliction.  The  major  team  members  within  the  GCS  include  the 


A  VO  (Air  Vehicle  Operator)  who  operates  the  UAV,  the  PLO  (Payload  Operator)  who  operates 
the  sensors,  and  the  DEMPC  (Data  Exploitation,  Mission  Planning,  and  Communications 
Operator)  who  is  responsible  for  mission  planning.  These  individuals  work  together  to 
accomplish  the  goal  of  navigating  the  UAV  to  a  position  to  take  reconnaissance  photos  of 
designated  targets.  Individual  team  members  have  access  to  information  about  the  UAV  flight 
system,  sensor  equipment,  and  the  surrounding  environment,  by  way  of  computer  displays,  hard 
copies,  and  communication  channels. 

Cooke  and  Shope  (in  review)  write  that  in  their  case,  the  STE  was  designed  to  serve  as  a 
flexible  task  environment  for  the  development  and  evaluation  of  measures  of  team  cognition.  It 
was  also  to  be  located  in  New  Mexico  State  University's  CERTT  laboratory.  This  overall  goal 
resulted  in  the  following  three  objectives:  1)  the  STE  should  facilitate  the  measurement  of  team 
cognition,  2)  the  STE  should  provide  a  realistic  task  environment,  and  3)  the  STE  should  provide 
an  experimenter-friendly  research  test-bed 

Their  resulting  prototype  consisted  of  seven  interconnected  systems  (two  for  each  of 
three  team  members  and  one  for  the  experimenter),  and  for  each  system,  a  representation  of  the 
screen  and  detailed  functional  specifications. 

C3STARS  The  Command,  Control,  Communications,  Simulation,  Training  and 
Research  System  (C3STARS)  facility  offers  the  opportunity  to  investigate  complex  decision 
making  among  interdependent  team  members  within  a  dynamic  and  realistic  setting.  The 
crewstations  and  scenarios  simulate  the  air  defense  mission  of  an  AW ACS  platform.  Realism  is 
achieved  through  the  functional  representation  of  equipment  and  displays,  experienced  personnel 
playing  the  role  of  simulation  pilots,  and  the  use  of  operational  scenarios. 

Closed  circuit  video  and  audio  stations  permit  experimenters  to  directly  observe  team 
interactions  and  remotely  record  all  communications  (computer,  visual  and  audio)  for  later 
analysis.  The  unique  simulations  integrate  hardware  and  software  resources,  data  collection  and 
analysis  systems,  verbal  communication  networks,  command  and  control  scenarios  and  team 
performance  measures. 

The  capability  of  the  facility  is  enhanced  by  connecting  the  crewstations  to  the  Advanced 
Distributed  Simulation  (ADS)  network  -  enabling  assets  at  other  DOD  facilities  to  be  integrated 
into  multi-force  simulation  exercises. 

The  communication  network  takes  advantage  of  advances  in  high  speed  digital  signal 
processing,  networking  and  communication  architectures  to  provide  a  communication  system 
that  will  grow  with  future  technology.  The  network  is  also  designed  to  support  synthetic  tasks 
requiring  voice  messages  on  low  cost  PC-based  communication  workstations.  The  system  can 
be  linked  into  the  internet  with  other  government  laboratories  and  universities. 

E-PASS.  The  Plight-Performance  Assessment  Simulator  (P-PASS)  is  an  inexpensive 
desktop  flight  simulator  designed  to  teach,  evaluate,  and  select  for  piloting  and  situation 
awareness  (SA)  skills  (O’Donnell  &  Moise,  1997,  O’Donnell  &  Moise,  in  review).  It  is  a  multi- 
dimentional,  theory  based,  dynamic  measurement  tool  that  can  generate  measures  of  SA, 
workload,  skill  level,  stress  training,  etc.  Its  data  can  be  validated  against  the  ultimate  criterion  - 
mission  performance. 

Current  scenarios  are  applicable  to  the  performance  of  reconnaissance,  surveillance,  air- 
to-air,  and  air-to-ground  missions  in  multiple  threat  environments,  and  range  from  minutes  to 
hours  in  length.  The  P-PASS  utilizes  realistic  aeromodels  that  currently  represent  the  dynamic 


characteristics  of  the  F-16,  T-38  and  T-1  aircraft.  The  F-PASS  also  has  the  capability  to  add  new 
aircraft  and  new  scenarios  to  the  system. 

Latest  Interoperability  Tests 

In  December  of  2001,  a  successful  test  was  run  linking  the  C3STARS,  the  UAV,  and  the 
FPASS  simulators.  The  C3STARS  and  UAV  simulators  utilized  T1  lines,  while  the  FPASS  used 
a  30K  baud  line.  While  connected,  all  3  of  the  systems  participated  in  an  exercise.  The  FPASS 
accurately  displayed  both  the  UAV  and  the  C3STARS  generated  entities.  The  C3STARS  saw  all 
entities  involved. 

As  an  example  of  how  these  varied  simulators  were  integrated,  the  UAV  found  ground 
threats,  and  this  data  was  sent  to  the  C3STARS.  The  C3STARS  then  used  this  data  to  direct 
friendly  forces,  including  both  FPASS  and  C3STARS  generated  entities. 

One  limitation  this  test  faced  was  the  fact  that  the  UAV  was  set  to  transmit  data  only.  It 
did  receive  data  from  the  C3STARS,  but  there  was  no  way  for  it  to  utilize  the  data.  Also,  voice 
communications  were  sent  over  phone  lines.  Interest  has  been  displayed  at  integrating  voice 
communication  into  the  network  in  the  future. 

In  March  of  2002,  a  separate  test  was  run,  this  time  successfully  connecting  the 
C3STARS  entities  and  the  AEDGE.  This  entailed  one  way  communication  from  the  C3STARS 
to  the  AEDGE.  Position,  heading,  altitude,  and  identification  data  were  all  communicated,  and 
all  information  was  accurate  with  the  exception  of  some  problems  with  the  heading  information. 

This  test  was  not  run  in  the  reverse  direction,  with  information  going  from  the  AEDGE  to 
the  C3STARS,  although  it  is  believed  that  it  would  likely  work.  Another  potential  advance 
would  involve  utilizing  the  AEDGE  agent  recommendation  information  and  sending  it  to  the 
C3STARS  operator. 

Summary 

These  efforts  have  shown  the  great  potential  that  exists  for  the  integration  of  simulators 
into  a  common  network,  one  that  can  be  utilized  not  only  for  training  purposes  but  also  for 
training  research.  Each  of  the  UAV,  EPASS,  and  C3STARS  was  developed  independently, 
without  plans  for  integration.  However,  tests  have  been  successfully  run  showing  them  working 
together,  generating  and  displaying  entities  from  each  of  the  various  systems  in  a  common 
battlespace.  Integration  barriers  are  being  taken  down,  allowing  for  interoperability  capabilities 
previously  undreamed  of.  This  will  allow  for  high  quality  collaborative  research,  where  the 
strengths  of  each  simulator  and  platform  is  put  to  use  in  a  complimentary  way.  The  creation 
of  this  infrastructure  is  more  than  a  demonstration  of  technology.  Eirst,  the  platforms  themselves 
offer  unique  capabilities,  regarding  scenario  realism,  experimental  control  and  performance 
measures  to  experimenters.  The  use  of  a  common  platform  allows  comparison  of  diverse 
programs  of  research,  each  focused  on  a  different  aspect  of  performance,  albeit  training,  interface 
technology,  information  distribution,  or  fatigue  countermeasures. 

Even  so,  why  connect  the  platforms?  Certainly,  the  distribution  and  portability  of  these 
platforms  have  self-evident  benefits  for  trainers — training  can  occur  among  distributed  trainees, 
deployed  in  remote  sites.  But  what  is  the  benefit  for  researchers?  Eirst,  there  is  great 
importance  in  performing  research  on  multi-team  system  performance.  Operational  teams  are 
often  assigned  ad  hoc,  with  team  members  and  teams  having  diverse  perspectives  and  little 


familiarity  with  each  other.  Distributed  team  research  can  capture  these  inherent  differences 
faced  by  DMT  teams.  Teams  in  different  locations  would  have  diverse  individuals,  trained  in 
diverse  locations,  with  different  curricula,  by  different  trainers — thus  capturing  relevant  and 
realistic  diversity  in  operations. 

In  addition,  operational  DMT  generates  questions  regarding  the  type  and  nature  of  joint 
mission  planning,  multi-team  coordination,  and  joint  debriefing  procedures.  DMT  teams 
encompass  diverse  teams,  such  as  cadres  of  fighter  pilots,  cargo  and  refueling  aircrew,  and 
various  command  and  control  platforms.  DMT  itself  does  not  lend  itself  as  well  to  experimental 
design.  DMT  resources  are  devoted  more  toward  training  in  itself,  subjects  /  resources  are 
difficult  to  procure,  and  experimental  manipulation  more  difficult  to  achieve,  in  a  context  rich  in 
confounding  variables  and  low  in  statistical  power.  The  connection  of  diverse  internet-based 
platforms  allow  university-based  researchers  to  investigate  questions  of  skill  acquisition  in  multi¬ 
team  context,  to  easily  manipulate  distribution  and  display  of  information  and  performance 
feedback,  and  to  study  processes  of  joint  mission  planning  and  debriefing.  Thus,  DMT-RNet 
will  serve  as  a  scientific  bridge  to  for  the  enhancement  of  operational  distributed  mission 
training. 
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